Universal Access To All Knowledge
Home Donate | Store | Blog | FAQ | Jobs | Volunteer Positions | Contact | Bios | Forums | Projects | Terms, Privacy, & Copyright
Search: Advanced Search
Anonymous User (login or join us)
Upload

Reply to this post | See parent post | Go Back
View Post [edit]

Poster: Administrator, Curator, or StaffJeff Kaplan Date: Jul 29, 2012 1:33pm
Forum: web Subject: Re: Archive appears sometimes, other days it won't

what is the url? the current owner of the domain may have put a robots.txt file.

Reply to this post
Reply [edit]

Poster: amnion Date: Jul 31, 2012 2:03pm
Forum: web Subject: Re: Archive appears sometimes, other days it won't

The site was www.tarotjunkie.com. It goes back and forth. Some days the site works on here and other times it doesn't.

Reply to this post
Reply [edit]

Poster: Administrator, Curator, or StaffJeff Kaplan Date: Jul 31, 2012 2:18pm
Forum: web Subject: Re: Archive appears sometimes, other days it won't

there is a robots.txt in place
http://www.tarotjunkie.com/robots.txt

Reply to this post
Reply [edit]

Poster: amnion Date: Aug 9, 2012 3:54pm
Forum: web Subject: Re: Archive appears sometimes, other days it won't

Wouldn't that only apply to the current versions of that site? What about the versions of the site where there were no robots.txt in place? The website has already been crawled, when it was okay to crawl it and archive it. The robots.txt just says "do not crawl this website". That would mean it can't be archived right now, at this moment. It doesn't seem fair that a new domain owner can put out a decree stating that this website, and all versions of this website from the past (which I did not own) are mine to control and must not be archived.

I am only trying to find the versions that have already been archived, which I know exist, because I've seen them.

Reply to this post
Reply [edit]

Poster: Administrator, Curator, or StaffJeff Kaplan Date: Aug 9, 2012 4:15pm
Forum: web Subject: Re: Archive appears sometimes, other days it won't

robots.txt blocks all archived captures for that url.

Reply to this post
Reply [edit]

Poster: amnion Date: Aug 9, 2012 5:30pm
Forum: web Subject: Re: Archive appears sometimes, other days it won't

How do we change that? Who do I need to talk to?